Decentralized Learning With Lazy and Approximate Dual Gradients

This paper develops algorithms for decentralized machine learning over a network, where data are distributed, computation is localized, and communication restricted between neighbors. A line of recent research in this area focuses on improving both complexities. The methods SSDA MSDA (Scaman et al., 2017) have optimal complexity when the objective smooth strongly convex, simple to derive. However, they require solving subproblem at each step, so required accuracy solutions total computational complexities uncertain. We propose new that instead subproblem, run warm-started Katyusha small, fixed number steps. In addition, previous information sufficiently useful, local rule will decide even skip round communication, leading extra savings. show our efficient provably reducing MSDA. numerical experiments, achieve significant reduction compared with state-of-the-art.